Throughout the specification, the expression "scene of a video program" (and variations thereon) denotes one or more consecutive frames of video or other image data (optionally with corresponding audio data). A video program can comprise one such scene or a sequence of at least two such scenes. Throughout the specification, the expression "scene of an audio program" (and variations thereon) denotes a single or multitrack analog or digital audio signal. An audio program can comprise one such scene or a sequence of at least two such scenes.
Throughout the specification, the term "clip" denotes data determining a scene of a video or audio program, the term "video clip" denotes data which determines a scene of a video program, and the term "audio clip" denotes data which determines a scene of an audio program.
Throughout the specification, the expression "display of a clip" denotes display of a representation of the clip. Also throughout the specification, the expression "realtime processing" (and variations thereon) denotes processing performed on video data, other image data, or audio data at the standard transmission rate of such data (e.g., the standard NTSC video rate of about 60 fields per second, or the standard
video rate of 50 fields per second, for video data).
Typically, an edited video program is determined by a sequence of video and audio clips, some of which may overlap along the program's timeline. Conventional systems for editing a video program typically employ a programmed computer to perform video editing operations in which the computer creates and manipulates clips of the video program. For example, U.S. Pat. No. 4,538,188, issued Aug. 27, 1990, to Barker, et al., describes one such system. The system of U.S. Pat. No. 4,538,188 employs a computer to control the display and manipulation of pictorial "labels," each representing an unedited scene of a video program, to assemble an edited video program from unedited video scene.
Other video editing systems have been proposed which include means for storing unedited video clips (on video tape or laser video disks), and computer means for controlling the display of selected unedited video scenes (or video frames) and generating an edit list in response to user-entered commands.
For example, U.S. Pat. No. 4,796,994, issued May 24, 1988, to Ettlinger, discloses a video editing system in which unedited video data is stored on video tape recorders or video disk players. A computer is programmed to control the video tape recorders and generate an edit list in response to user commands. The computer displays a sequence of menus which prompt the user to perform various editing operations (such as displaying desired frames of the unedited video scene, adding edits to an edit list, and playing back the sequence of video scenes defined by the edit list).
For another example, U.S. Pat. No. 4,754,352, issued Jun. 28, 1988, to Duffy, describes a computer-based video editing system in which unedited video data is stored on video disk players. After generating an edit list, the user may command the system to splice scenes in accordance with the edit list and play back an edited video program. By using a control console with a control buttons and a control dial, the user may command the system to display individual frames of the stored scenes, or "roll" one or more scenes (or an edited sequence of scenes) in forward or reverse motion.
Conventional video editing systems generate clips of a video program, each clip consisting of "content" data identifying the beginning and end ("in" and "out") frames of a stored segment of video data to be included in an edited program (and sometimes also beginning and end points of a corresponding audio soundtrack). Such "content" clips are manipulated in computer software (which may have a graphical timeline display representation) to assemble an edit list which defines the edited program. Conventional edit controllers include computer software for controlling video (and audio) storage devices to selectively copy stored raw video and audio scenes of unedited programs, in a sequence determined by the edit list, to an edited program storage means.
However, it is also desirable during video editing to specify "processing" data which specifies various filters or other processing operations to be applied to an individual scene (or one or more frames within a scene) of an edited program. Consider for example, use of computer-implemented video editing to determine a video program edit list which is in turn used to generate (photographically) a film print whose frames correspond to video frames of the edited video program. In this case, the editor who prepares the edit list typically also prepares elaborate notes about the desired processing operations. These notes are passed on to the film processor, who uses them during preparation of the final film print. The editor cannot see the results of the processing operations that he (she) specifies until after the film processor prepares the final print.
In another example, computer-implemented video editing is employed to generate (electronically) an edited video program. In this case, computer software employed to prepare an edit list (typically in response to user manipulation of graphically displayed clips) and to control recording of video scenes (specified by the edit list) in a program storage device in the sequence specified by the edit list, can also control one or more processing devices for processing frames of the edited program as they are sequentially read out from the program storage device (the processed frames output from the processing device can be displayed to enable the editor to preview the edited program, or can be written back into the program storage device or some other storage device). Thus, processing can be performed at the same time as content editing, to give the editor rapid feedback. Conventional processing devices sometimes include internal memory for storing processing parameters, but they do not have any knowledge of when during a program (i.e., to which scenes and frames of scenes) a given set of processing parameters should be applied. An external controller can be used to trigger a processing device based on the program time (e.g., the processing device can store a control signal from the controller which instructs the processing device to execute "effect X" at "2 minutes and 30 seconds" after the start of the program), but external controllers have not been used to instruct processing devices to perform specified operations on specific scenes (or frames within scenes) regardless of the times at which such scenes are received (in sequence) by the processing device.
A disadvantage of such conventional control of a processing device with an external controller is that, if the editor rearranges the order of scenes in a program (and thus the sequence of data to be read out from the program storage device) after the controller has instructed the processing device to apply a set of processing parameters at a given program time, the processing device will apply the parameters to the wrong frames as it receives the frames from the program storage device.
Because the devices for keeping track of program content (e.g., an edit controller programmed with software for performing edit list generation and for copying stored frames of data to a "program storage device") are separate from those for performing processing on frames of a video program, the operator must do most of the synchronization between the devices in his (her) head, or by keeping meticulous manual notes. This would not be a significant problem if a program were edited only once, and in strict chronological order (in which case the processing could be manually applied once to each scene, as the data defining the scene is copied from a raw scene storage device into the program storage device). Unfortunately, in the real world, programs are often edited and re-edited many times, sometimes by different individuals. Furthermore, recent advances in "non-linear" editing systems (which enable scenes of a program to be edited in non-chronological order) have increased the ease with which program content can be changed, thus increasing the need to remember the processing parameters to be applied to each scene. Without a fully integrated system for remembering both program content and the corresponding processing parameters, an edited program will either take longer to put together (because the operator must remember and reset all of the processing parameters each time he or she makes a change to the "content" of the program), or will have low quality (because the processing device will apply the wrong processing parameters to one or more of the scenes, so that the processing will not match from one scene in the desired manner).
Several techniques have been proposed for sending processing cue signals from an edit controller to a processing device. Typically, the edit controller does not "know" which processing operation will be performed in response to any of the cue signals that it sends. Typically, the user must pre-program the processing device to perform several processing operations (e.g., "Process #1" and "Process #2"), and the edit controller sends cue signals at desired times to the processing device to trigger the next processing operation (e.g., the edit controller "knows" only that a cue signal will trigger "Process #2"). Cue signal generation by the edit controller may be loosely coupled with content parameters (contained in a clip) "known" to the edit controller.
A number of "non-realtime" video editing systems are commercially available. These systems can generate edit lists by manipulating "clips" which represent underlying raw video data, but (unlike "realtime" video editing systems) are incapable of applying necessary processing such as picture compression or other digital video effects to desired stored raw video scenes (or frames of scenes) and assembling the processed scenes into an edited video program, at standard video rate (e.g., about 30 frames per second for NTSC video). In the context of "non-realtime" video editing systems, it is known to include, in the video clips, frame-by-frame calls to one or more filter programs which apply video or audio effects. This method is implemented, for example, in the non-realtime video editing software known as "Adobe Premiere.TM." software, available from Adobe Systems Incorporated. However, non-realtime systems of this type do not allow the user to see the effects of the filtering operations unless the raw video scenes represented by the clips are rendered in non-realtime to a video storage device (which could include a disk storage means), which can be a very slow operation depending on how many clips are simultaneously playing (i.e., overlapping in time) and the computational effort that is required for the processing required.
Conventional realtime video editing methods and systems have not stored processing data in "content" data-containing clips. Nor have they employed dedicated processing hardware for processing video data (representing scenes of a program) in realtime in accordance with the processing data, while the video data is copied from storage devices to assemble the program in realtime (in accordance with the content data in a sequence of the clips).